Diskify Your Mind: build a 64DD softpatch disk from an "AI Shogi 3" ROM.

.: Introduction :.
  AI Shogi 3 is unique in the N64 library.  It is the only non-expansion disk title that has a full Leo library--the library that drives 64DD support.  It also uses an earlier version than any other cart or disk release (B014A25 vs B014A26).  Normally the disk thread is not started so it will ignore the drive, but linking two functions via a cheat device allows the retail game to read filetable data from the disk.
  That's right.  With a GameShark you can softpatch data stored on disk to the original cartridge.  This works with both retail and development drives.

  Diskify creates that softpatch from the data stored on a cartridge.  AI Shogi 3 doesn't enforce disk IDs so it can reside alongside other disk data, such as the Dezaemon 3D expansion, or it can sit in a retail title's MFS directory (so long as it is not fragmented at all in any way, shape, or form).
  Normally you'd be limited to the original filesizes given in the filetable, but part of what Diskify does is create a replacement directory at the start of data.  The "EX Disk Filetable" codes load this over the original table.


.: Using Diskify :.
  Diskify is a 64bit Windows application.

  Opening Diskify will start the GUI.  It can also be used on the command line if you provide a filename or options (try: diskify.exe --help).  Hovering over most options in the GUI will display short help popups.

  Before saving is enabled you must load an "AI Shogi 3" ROM.  This is the source data used for creating the softpatch.  The ROM may be native format (big endian) or little endian.  Patched games can be used in most cases, but keep in mind only the images will be stored disk, not any layout or code changes.  The text window will display success or failure opening the file and the filename will be printed next to the [Load N64 ROM] button.
  
  [Output Settings] changes the type of file generated.  It will generate either a retail 64DD disk image, a development disk image, or output just the expansion data.  The disk images are usable in emulators that support the 64DD or can be written back to hardware.  Only type 0 disks are created by this tool.  [Disk Options] controls the region (locked to Japanese due to early lib restrictions) and an option to combine this disk with Dezaemon 3D expansion data.
  One use of the expansion data is to load it to the RAM area of a retail disk.  If the target game uses MFS be absolutely certain it does not fragment the file in any way.  One way is to mark the blocks reserved or put write-protect blocks on them.
  
  When ready, press the [Save Output] button and provide an output filename.  If a disk mode is selected, it will output a disk image in .ndd format.  If [Data Only] was selected, the default extension is .EXDAT.
  
  At the bottom of the window is a toggle to reveal the same GameShark codes listed here.


.: Disk Support GameShark Codes :.
  In order for an original cartridge to run the expansion data you will need a GameShark / Pro Action Replay v3.2 or v3.3 to start the disk thread and enable the filetable replacement extension.  Minimum support requires "Disk Support" and "EX Disk Filetable".

  These use at-boot codes (v3.2 & v3.3).  There's a limit of 50 of these.  Codes are written at certain system interrupts, so at-boot writes not only preserve code space but guarantee the ASM changes took effect during the boot process.
"Disk Support"
F1061BA4 0801
F1061BA6 212C
F104856C 0801
F104856E 225C
F0038664 008C
F0038684 008C
F0038690 00AC
F104BFFC 0801
F104BFFE 3045
F104C000 0000
F10374CC 3C04
F10374CE 8040
F10374D4 3C05
F10374D6 0020

"EX Disk Filetable"
F1025C48 3C07
F1025C4A 8008
F1025C4C 2404
F1025C4E 000A
F1025C50 2405
F1025C54 24E7
F1025C56 DF10
F1025C58 0801
F1025C5A 2326
F1025C5C 2406
F1025C5E 0840
F1028254 0C00
F1028256 9712

  It can also be used with retail disks from RAM space if you change the base LBA from 000A/0022 to match where the data is.  So yes, you can make a AISHOGI3 file, stick it as a file in another game's MFS directory, and use these GameShark codes to set where to look:
F1025C4E ****
F104C13E ****
F104C19A ****

.: Optional :.
  To enable the debugger's callback (press L whenever to open it):
F10286A0 0801
F10286A2 31BF

  The special enable code for 64DD titles (CC000000 0000) is not needed unless code generator is on.  That copies the hook to 800002C0 and sets a jump there, preventing the disk ID from overwriting the normal hook's code at 800001A0-800001C0.
"(M)"
CC000000 0000


.: What These Do :.
  Let's walk through these codes one-by-one.
80061BA4: 0801212C
  This hooks 800484B0() after PI initialized.  This is the disk initialization function, starting both the Leo threads and the high-level interface unique to this game used for read/seek/init requests.

8004856C: 0801225C
  Calling 80048970() sends a message to the disk manager above to store the disk ID and start listening for file requests.

8004BFFC: 08013045 0000
  This redirects the function to decompress data from cart to a function with the exact same interface used to decompress data from disk.

80038664: 008C
80038684: 008C
80038690: 00AC
  The difference between cart and disk file decompression is that the disks require compressed data all be loaded to memory before unpacking it.  There's also an arbitrary difference where compressed files are not expected to be larger than 0x10000 bytes.
  There are a total of ~24 full-screen backgrounds that exceed this limit and will fail to decompress correctly.  These three codes remove that limit by saving and loading 4 byte words instead of 2 byte halfwords.

800374CC: 3C048040
800374D4: 3C050020
  Now that the limit is removed, about seven of those files require more allocated memory to store both the compressed and decompressed copies of data in memory simultaneously, especially in hi-res configuration when the buffer is halved.  The game doesn't use the 8MB Expansion Pak, so these two codes set a much larger allocation buffer in unused memory.
  If the images were split in strips these two fixes wouldn't be needed.


.: Manifesto of Glorious Speculation :.
  The case being made here--as ridiculous as it may sound--is that AI Shogi 3 started its life as a 64DD disk and was quickly ported to cartridge.
  
  What makes it exceptional is it's the only cartridge in the library with 64DD LBA tables and Leo libs--aka actual 64DD support--that isn't designed for (or as) an expansion disk.  The complete set of titles is rather short:
Dezaemon 3D	1998/6/26
	CDZJ
F-Zero X	1998/7/14
	CFZE, NFZG, CFZJ, NFZP, iQue (code present but unlinked)
Pocket Monsters Stadium	1998/8/1
	CPSJ
* Pocket Monsters Stadium 2 / Pokemon Stadium	1999/4/30; org. cart+exp compiled as one product
	CP2J, NPOD, NPOE*, NPOF, NPOI, NPOP*, NPOS
The Legend of Zelda - Ocarina of Time	1998/11/21
	CZLE*, CZLJ* (Any other release with the two disk files in the FS)
Mario Party	1998/12/18
	CLBE, CLBJ, NLBP
* Mario Party 2	1999/12/17; org. cart+exp compiled as one product
	NMWE, NMWJ, NMWP
AI Shougi 3	1998/9/21 internal, 1998/12/18 release
	NS3J

  Note games like Ogre Battle 64 that originally were developed as cart+exp no longer have any disk code.  A subset of about 14 titles will link a cart interrupt handler during system initialization if a drive is found but lack the libs needed for support--a quirk in a particular build of OS 2.0J.

  There's three types of expansions:
  Stadium, Mario Party, and F-Zero X expansion disks all "reboot".  They replace the cart game code with updated code on disk.  It's more accurate to say they're disk games that use a few things on their original cartridges.  There isn't much reason they can't exclusively be a cartridge or a disk, and that's exactly what happened 66% of the time.
  The reboot portion of that is important.  These include code that decrypts a second decrypter for the special entrypoint in the disk's executable using the address of the decryption function itself, then stops all threads, simulates an NMI, and runs the decrypted disk code.  It's very specialized code and necessary (outside the encryption).  AI Shogi 3 doesn't have it so it absolutely isn't this kind of title.
  
  Ocarina fits our usual understanding of an expansion.  The disk contains replacement data and has lists of replacements.  When a part of the game loads data, it checks if the file it wants is marked in the table and replaces if necessary.  The cartridge solely drives the game, although the disk has the capability of patching the executable.
  Replacement is only practical when tables are provided.  AI Shogi 3 uses its existing table in memory when loading files starting at LBA 000A/0022, rather implying it expects to read its normal data from there (as much of a placeholder address as that may be).
  
  Dezaemon 3D is storage.  Its ROM area can have read-only game data, but the RAM area becomes the only storage method and it unlocks more features in the editor.  The disk only provides data and storage; the entire runtime and interface is loaded from cartridge.
  Interestingly, write operations in AI Shogi 3's disk manager are stubbed; you can use the lib functions of course, but there isn't a friendly way to do it.

  Then there's AI Shogi 3.
  The Leo library wasn't accidentally compiled in.  There is a custom-made higher-level interface that ties into a custom-made disk request manager.  The requests are sent using functions with the exact same argument sets as the cart request functions.
  It's also no accidental include from another title in development via debug includes as happens so often.  The debug menu is used among a series of A.I developed titles, each iteration rolling in resources from the last.  For example, titles after Densha de Go include its VRS test and the libs needed to run it, all have rumble and mempak tests, etc.  None of their other releases have disk functions and there is no tested disk function in these debug menus.
  The internal date of 1998/9/21 is odd in its own right.  It's linked against libleo from the short-lived OS 2.0I (B014A25), released ~December 1997, and all other titles are linked against OS 2.0J or higher (B014A26)-- yet it's released after the other expansion titles?  There isn't much here; it's a 2D game similar to the PSX prequel released the year before.  A nine month development cycle for a 4.1MB game implies a rework.
  
  There's quite a bit in duplicate in the high-level interface.  There's both a cart and disk decompression function, iterator functions, both raw and compressed filetable load functions (though raw isn't used), raw data loaders--and the interface to all of these is identical.
  In fact, the only significant change to revert it to a pure disk title is to replace the audio libs with their 64DD counterparts.  These load the entire sample table from disk to memory, accepting some different arguments to make it happen, or alternately using samples stored in the 64DD IPL.  Otherwise, to load code or data simply replace the cart function pointers with their disk equivalents.
  That also raises a question.  There is one line of music and corresponding soundbank intended for the attract video, but both are empty placeholders.  Was it written using IPL audio sources and never backported?
  
  Another more subtle design point is that all the resources loaded in a given menu or during gameplay are organized so they're loaded sequentially.  This plays a large part in how short load times are.  Reading from disk is quite fast; seeking is not.  Cartridge games are random-access and don't need to fuss over this.  You'll find their assets are usually in alphabetical order, almost as if somebody copied the output from DIR ;*)
  
  So if you buy into this wild speculation, the game was originally built as a pure disk, not an experimental cart or an expansion.  There isn't support for any other peripheral included (like the modem) and it didn't write files (at least in this iteration), it just ran the game from a disk.  Maybe they were either scared off by the realization the distribution method wasn't going to be at kiosks like FDS or the Nintendo Power carts, nor via download like Satellaview; maybe the disk version had a serious bug that couldn't be worked out; or maybe there was a miscommunication and the N64 version wasn't supposed to be developed as a disk in the first place.  At any rate, it's a bit of an odd bird.

-Zoinkity
